The present invention relates generally to computer graphics, and more particularly to a method for extracting textures from a sequence of images.
Texture-mapping is a well known method for adding detail to renderings of computer graphic scenes. During texture-mapping, textures are applied to a graphics model. The model typically includes a set of 3D points and a specification of the edges, surfaces, or volumes that connect the points. If the rendering is volumetric, e.g., the rendering represents a solid object and not just its surface, then the textures can be in 3D. Texture-mapping gives the illusion of greater apparent detail than present in the model""s geometry. If the textures are extracted from photographs or images, the rendered images can appear quite realistic. Textures can be in the form of texture-maps, i.e., data, or texture-functions, i.e., procedures.
In general, textures and images are sparse samplings of a light field, typically, the reflected light from a surface. Digital images, including individual video frames, are fairly low-resolution samplings of reflectance. However, if there is motion in a sequence of images, then every image gives a slightly different sampling of the light field, and this information can be integrated over time to give a much denser sampling, also known as a super-resolution texture.
It is desired to provide super-resolution textures from a low-resolution sequence of images.
The method according to the invention extracts texture-maps and texture-functions, generally xe2x80x9ctextures,xe2x80x9d from a sequence of images and image-to-image correspondences. The extracted textures have a higher resolution and finer detail than the sequence of images from which they are extracted. If every image is annotated with control parameters, then the invention produces a procedure that generates super-resolution textures as a function of these parameters. This enables dynamic textures for surfaces or volumes that undergo appearance change, for example, for skin that smooths and pales when stretched and wrinkles and flushes when relaxed. These control parameters can be determined from the correspondences themselves.
In the following text the term xe2x80x9cimagexe2x80x9d will be used for 2D or 3D arrays of data and xe2x80x9cvideoxe2x80x9d will be used for time-series of such arrays.
More specifically, the invention provides a method that constructs a super-resolution texture from a sequence of images of a non-rigid three-dimensional object. The object need not be non-rigid. A shape of the object is represented as a matrix of 3D points. A basis set of possible deformations of the object is represented as a matrix of displacements of the points. The matrices of 3D points and displacements form a model of the object in the video and its possible deformations. The points in the model are connected to form a triangle or tetrahedron mesh, depending on whether the images are 2D or 3D. For each image, a set of correspondences between the points in the model and the object in the image is formed. The correspondences are used to map the mesh into each image as a texture mesh. Each mesh, and the image texture the mesh covers, is warped to a common coordinate system, resulting in a texture that appears to be a deformation of the original image. The warp is done with super-sampling so that the resulting texture has many more pixels than the original image. The warped and super-sampled textures are averaged to produce a static super-sampled texture of the object in the image. Various weighted averages of the warped and super-sampled textures produce dynamic textures that can vary according to the deformation and/or pose of the object.